Example: Hyperparameter tuning¶
This example shows an advanced example on how to optimize your model's hyperparameters for multi-metric runs.
Import the breast cancer dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict whether a patient has breast cancer or not.
Load the data¶
In [1]:
Copied!
# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier
# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier
In [2]:
Copied!
# Load the data
X, y = load_breast_cancer(return_X_y=True)
# Load the data
X, y = load_breast_cancer(return_X_y=True)
Run the pipeline¶
In [3]:
Copied!
# Initialize atom
atom = ATOMClassifier(X, y, verbose=2, random_state=1)
# Initialize atom
atom = ATOMClassifier(X, y, verbose=2, random_state=1)
<< ================== ATOM ================== >> Algorithm task: binary classification. Dataset stats ==================== >> Shape: (569, 31) Memory: 138.96 kB Scaled: False Outlier values: 169 (1.2%) ------------------------------------- Train set size: 456 Test set size: 113 ------------------------------------- | | dataset | train | test | | - | ----------- | ----------- | ----------- | | 0 | 212 (1.0) | 170 (1.0) | 42 (1.0) | | 1 | 357 (1.7) | 286 (1.7) | 71 (1.7) |
In [4]:
Copied!
# Train a LogisticRegression model on two metrics
atom.run(models="LR", metric=["f1", "ap"], n_trials=10)
# Train a LogisticRegression model on two metrics
atom.run(models="LR", metric=["f1", "ap"], n_trials=10)
Training ========================= >> Models: LR Metric: f1, average_precision Running hyperparameter tuning for LogisticRegression... | trial | penalty | C | solver | max_iter | l1_ratio | f1 | best_f1 | average_precision | best_average_precision | time_trial | time_ht | state | | ----- | ------- | ------- | ------- | -------- | -------- | ------- | ------- | ----------------- | ---------------------- | ---------- | ------- | -------- | | 0 | l2 | 0.0054 | saga | 480 | --- | 0.9744 | 0.9744 | 0.9991 | 0.9991 | 0.017s | 0.017s | COMPLETE | | 1 | l2 | 0.122 | saga | 380 | --- | 0.9828 | 0.9828 | 0.9859 | 0.9991 | 0.024s | 0.041s | COMPLETE | | 2 | l2 | 0.0071 | sag | 720 | --- | 0.9744 | 0.9828 | 0.9947 | 0.9991 | 0.014s | 0.055s | COMPLETE | | 3 | l2 | 87.9641 | libli.. | 920 | --- | 0.9464 | 0.9828 | 0.9591 | 0.9991 | 0.013s | 0.068s | COMPLETE | | 4 | none | --- | sag | 630 | --- | 0.9744 | 0.9828 | 0.9631 | 0.9991 | 0.071s | 0.139s | COMPLETE | | 5 | l2 | 0.0018 | sag | 920 | --- | 0.9256 | 0.9828 | 0.9921 | 0.9991 | 0.015s | 0.154s | COMPLETE | | 6 | l2 | 43.4053 | sag | 780 | --- | 0.9483 | 0.9828 | 0.9686 | 0.9991 | 0.085s | 0.239s | COMPLETE | | 7 | l2 | 2.0759 | libli.. | 470 | --- | 0.9655 | 0.9828 | 0.992 | 0.9991 | 0.013s | 0.252s | COMPLETE | | 8 | l2 | 0.043 | sag | 110 | --- | 0.9744 | 0.9828 | 0.9971 | 0.9991 | 0.014s | 0.267s | COMPLETE | | 9 | l2 | 46.0233 | saga | 740 | --- | 0.9825 | 0.9828 | 0.988 | 0.9991 | 0.086s | 0.353s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 0 Best parameters: --> penalty: l2 --> C: 0.0054 --> solver: saga --> max_iter: 480 Best evaluation --> f1: 0.9744 average_precision: 0.9991 Time elapsed: 0.353s Fit --------------------------------------------- Train evaluation --> f1: 0.963 average_precision: 0.9941 Test evaluation --> f1: 0.966 average_precision: 0.9962 Time elapsed: 0.026s ------------------------------------------------- Total time: 0.379s Final results ==================== >> Total time: 0.397s ------------------------------------- LogisticRegression --> f1: 0.966 average_precision: 0.9962
In [5]:
Copied!
# For multi-metric runs, the selected best trial is the first in the Pareto front
atom.lr.best_trial
# For multi-metric runs, the selected best trial is the first in the Pareto front
atom.lr.best_trial
Out[5]:
FrozenTrial(number=0, values=[0.9743589743589743, 0.9990764494418141], datetime_start=datetime.datetime(2022, 9, 22, 10, 37, 10, 64858), datetime_complete=datetime.datetime(2022, 9, 22, 10, 37, 10, 81874), params={'penalty': 'l2', 'C': 0.005417257182767969, 'solver': 'saga', 'max_iter': 480, 'l1_ratio': 0.7000000000000001}, distributions={'penalty': CategoricalDistribution(choices=('l1', 'l2', 'elasticnet', 'none')), 'C': FloatDistribution(high=100.0, log=True, low=0.001, step=None), 'solver': CategoricalDistribution(choices=('lbfgs', 'newton-cg', 'liblinear', 'sag', 'saga')), 'max_iter': IntDistribution(high=1000, log=False, low=100, step=10), 'l1_ratio': FloatDistribution(high=1.0, log=False, low=0.0, step=0.1)}, user_attrs={'params': {'penalty': 'l2', 'C': 0.0054, 'solver': 'saga', 'max_iter': 480}, 'estimator': LogisticRegression(C=0.0054, max_iter=480, n_jobs=1, random_state=1,
solver='saga')}, system_attrs={'nsga2:generation': 0}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE, value=None)
In [6]:
Copied!
# Use the plot_pareto_front to select a better option
from optuna.visualization import plot_pareto_front
plot_pareto_front(atom.lr.study)
# Use the plot_pareto_front to select a better option
from optuna.visualization import plot_pareto_front
plot_pareto_front(atom.lr.study)
In [7]:
Copied!
# If you are unhappy with the results, it's possible to conitnue the study
atom.lr.hyperparameter_tuning(n_trials=5)
# If you are unhappy with the results, it's possible to conitnue the study
atom.lr.hyperparameter_tuning(n_trials=5)
Running hyperparameter tuning for LogisticRegression... | trial | penalty | C | solver | max_iter | l1_ratio | f1 | best_f1 | average_precision | best_average_precision | time_trial | time_ht | state | | ----- | ------- | ------- | ------- | -------- | -------- | ------- | ------- | ----------------- | ---------------------- | ---------- | ------- | -------- | | 10 | l2 | 19.959 | libli.. | 630 | --- | 0.9655 | 0.9828 | 0.9737 | 0.9991 | 0.021s | 0.373s | COMPLETE | | 11 | l2 | 10.8524 | newto.. | 150 | --- | 0.9735 | 0.9828 | 0.996 | 0.9991 | 0.018s | 0.392s | COMPLETE | | 12 | l2 | 0.6289 | libli.. | 320 | --- | 0.9821 | 0.9828 | 1.0 | 1.0 | 0.016s | 0.407s | COMPLETE | | 13 | l2 | 0.0158 | libli.. | 160 | --- | 0.9913 | 0.9913 | 0.9994 | 1.0 | 0.016s | 0.423s | COMPLETE | | 14 | l2 | 0.792 | libli.. | 160 | --- | 0.9744 | 0.9913 | 0.9956 | 1.0 | 0.016s | 0.438s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 12 Best parameters: --> penalty: l2 --> C: 0.6289 --> solver: liblinear --> max_iter: 320 Best evaluation --> f1: 0.9821 average_precision: 1.0 Time elapsed: 0.438s
In [8]:
Copied!
# The trials attribute gives an overview of the trial results
atom.lr.trials
# The trials attribute gives an overview of the trial results
atom.lr.trials
Out[8]:
| params | estimator | score | time_trial | time_ht | |
|---|---|---|---|---|---|
| trial | |||||
| 0 | {'penalty': 'l2', 'C': 0.0054, 'solver': 'saga... | LogisticRegression(C=0.0054, max_iter=480, n_j... | [0.9743589743589743, 0.9990764494418141] | 0.017016 | 0.017016 |
| 1 | {'penalty': 'l2', 'C': 0.122, 'solver': 'saga'... | LogisticRegression(C=0.122, max_iter=380, n_jo... | [0.9827586206896551, 0.9858721483975268] | 0.024022 | 0.041038 |
| 2 | {'penalty': 'l2', 'C': 0.0071, 'solver': 'sag'... | LogisticRegression(C=0.0071, max_iter=720, n_j... | [0.9743589743589743, 0.9947192281078494] | 0.014012 | 0.05505 |
| 3 | {'penalty': 'l2', 'C': 87.9641, 'solver': 'lib... | LogisticRegression(C=87.9641, max_iter=920, n_... | [0.9464285714285715, 0.9590727963935599] | 0.013179 | 0.068229 |
| 4 | {'penalty': 'none', 'solver': 'sag', 'max_iter... | LogisticRegression(max_iter=630, n_jobs=1, pen... | [0.9743589743589743, 0.9631035970079513] | 0.071064 | 0.139293 |
| 5 | {'penalty': 'l2', 'C': 0.0018, 'solver': 'sag'... | LogisticRegression(C=0.0018, max_iter=920, n_j... | [0.9256198347107438, 0.9921466959495802] | 0.015013 | 0.154306 |
| 6 | {'penalty': 'l2', 'C': 43.4053, 'solver': 'sag... | LogisticRegression(C=43.4053, max_iter=780, n_... | [0.9482758620689654, 0.9685587859139686] | 0.085077 | 0.239383 |
| 7 | {'penalty': 'l2', 'C': 2.0759, 'solver': 'libl... | LogisticRegression(C=2.0759, max_iter=470, n_j... | [0.9655172413793103, 0.9920077193219741] | 0.013012 | 0.252395 |
| 8 | {'penalty': 'l2', 'C': 0.043, 'solver': 'sag',... | LogisticRegression(C=0.043, max_iter=110, n_jo... | [0.9743589743589743, 0.9970693113085392] | 0.014181 | 0.266576 |
| 9 | {'penalty': 'l2', 'C': 46.0233, 'solver': 'sag... | LogisticRegression(C=46.0233, max_iter=740, n_... | [0.9824561403508771, 0.9879894597165679] | 0.086391 | 0.352967 |
| 10 | {'penalty': 'l2', 'C': 19.959, 'solver': 'libl... | LogisticRegression(C=19.959, max_iter=630, n_j... | [0.9655172413793103, 0.9736590712255391] | 0.020528 | 0.373495 |
| 11 | {'penalty': 'l2', 'C': 10.8524, 'solver': 'new... | LogisticRegression(C=10.8524, max_iter=150, n_... | [0.9734513274336283, 0.9959569642562589] | 0.018023 | 0.391518 |
| 12 | {'penalty': 'l2', 'C': 0.6289, 'solver': 'libl... | LogisticRegression(C=0.6289, max_iter=320, n_j... | [0.9821428571428572, 1.0000000000000002] | 0.015626 | 0.407144 |
| 13 | {'penalty': 'l2', 'C': 0.0158, 'solver': 'libl... | LogisticRegression(C=0.0158, max_iter=160, n_j... | [0.9913043478260869, 0.999389732649834] | 0.01563 | 0.422774 |
| 14 | {'penalty': 'l2', 'C': 0.792, 'solver': 'libli... | LogisticRegression(C=0.792, max_iter=160, n_jo... | [0.9743589743589743, 0.9955890815002342] | 0.015627 | 0.438401 |
In [9]:
Copied!
# Select a custom best trial...
atom.lr.best_trial = 2
# ...and check that the best parameters are now those in the selected trial
atom.lr.best_params
# Select a custom best trial...
atom.lr.best_trial = 2
# ...and check that the best parameters are now those in the selected trial
atom.lr.best_params
Out[9]:
{'penalty': 'l2', 'C': 0.0071, 'solver': 'sag', 'max_iter': 720}
In [10]:
Copied!
# Lastly, fit the model on the complete training set
# using the new combination of hyperparameters
atom.lr.fit()
# Lastly, fit the model on the complete training set
# using the new combination of hyperparameters
atom.lr.fit()
Fit --------------------------------------------- Train evaluation --> f1: 0.963 average_precision: 0.9941 Test evaluation --> f1: 0.966 average_precision: 0.9962 Time elapsed: 0.058s
Analyze the results¶
In [11]:
Copied!
atom.plot_trials()
atom.plot_trials()